Navigating visual data associated with a point of interest

ABSTRACT

Systems and methods for navigating an imagery graph are provided. In some aspects, a first image is provided for display, where the first image corresponds to a first image node within an imagery graph, where the imagery graph comprises image nodes corresponding to images from a plurality of different imagery types, and where each image node in the imagery graph is associated with geospatial data. An indication of a selection of a predetermined region within the first image is received, where the predetermined region is associated with a position in the first image that corresponds to geospatial data associated a second image node within the imagery graph. A second image corresponding to the second image node is provided for display in response to the indication of the selection of the predetermined region.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims the benefit of priority under 35 U.S.C.§119 from U.S. Provisional Patent Application Ser. No. 61/712,729entitled “NAVIGATING VISUAL DATA ASSOCIATED WITH A POINT OF INTEREST,”filed on Oct. 11, 2012, the disclosure of which is hereby incorporatedby reference in its entirety for all purposes.

BACKGROUND

The subject technology generally relates to visual data processing and,in particular, relates to navigating visual data associated with a pointof interest.

Images of points of interest (e.g., landmarks, buildings, or naturalwonders) are available on the Internet or in various data stores. Theimages may include different types of imagery (e.g., street viewpanoramas, aerial images, satellite images, user images from photographsharing services, or images from videos), as well as different sets ofimagery having the same type. However, the different types of imagery ofthese points of interest are oftentimes separate and disjoint from oneanother. As a result, an end-user of a client computing device may beable to see multiple views of the point of interest via the clientcomputing device but may have difficulty navigating between differentimagery sets and understanding the images of a specific point ofinterest within different imagery sets (e.g., street view images of theEmpire State Building and user images of the Empire State Building) areinterrelated with one another in three-dimensional space. As theforegoing illustrates, a new approach for combining and navigatingbetween multiple imagery sets including visual data associated with aspecific point of interest may be desirable.

SUMMARY

The disclosed subject matter relates to a computer-implemented methodfor navigating an imagery graph. The method includes providing a firstimage for display, where the first image corresponds to a first imagenode within an imagery graph, where the imagery graph comprises imagenodes corresponding to images from a plurality of different imagerytypes, and where each image node in the imagery graph is associated withgeospatial data. The method includes receiving an indication of aselection of a predetermined region within the first image, where thepredetermined region is associated with a position in the first imagethat corresponds to geospatial data associated a second image nodewithin the imagery graph. The method includes providing a second imagecorresponding to the second image node for display in response to theindication of the selection of the predetermined region.

The disclosed subject matter further relates to a computer-readablemedium. The computer-readable medium includes instructions that, whenexecuted by one or more computers, cause the one or more computers toimplement a method for providing an image for display. The instructionsinclude code for providing a first image for display, where the firstimage corresponds to a first image node within an imagery graph, wherethe imagery graph comprises plural image nodes, and where each imagenode in the imagery graph is associated with geospatial data. Theinstructions include code for receiving an indication of a selection ofone or more pixels within the first image, where the one or more pixelsare associated with a position in the first image that corresponds togeospatial data associated a second image node within the imagery graph.The instructions include code for providing a second image correspondingto the second image node for display in response to the indication ofthe selection of the one or more pixels.

The disclosed subject matter further relates to a system. The systemincludes one or more processors. The system also includes a memory. Thememory includes one or more data structures representing a unifiedimagery graph. The unified imagery graph includes image nodescorresponding to images from a plurality of different imagery types,where each image node in the unified imagery graph is associated with ageographic location, a viewpoint orientation, and a field of view. Thememory includes instructions. The instructions include code forproviding a first image for display, where the first image correspondsto a first image node within the unified imagery graph, and where thefirst image has a first imagery type. The instructions include code forreceiving an indication of a selection of a predetermined region withinthe first image, where the predetermined region is associated with aposition in the first image that corresponds to a geographic location, aviewpoint orientation, or a field of view associated a second image nodewithin the unified imagery graph. The instructions include code forproviding a second image corresponding to the second image node fordisplay in response to the indication of the selection of thepredetermined region, where the second image has a second imagery typedifferent from the first imagery type.

It is understood that other configurations of the subject technologywill become readily apparent to those skilled in the art from thefollowing detailed description, wherein various configurations of thesubject technology are shown and described by way of illustration. Aswill be realized, the subject technology is capable of other anddifferent configurations and its several details are capable ofmodification in various other respects, all without departing from thescope of the subject technology. Accordingly, the drawings and detaileddescription are to be regarded as illustrative in nature and not asrestrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

The features of the subject technology are set forth in the appendedclaims. However, for purpose of explanation, several aspects of thedisclosed subject matter are set forth in the following figures.

FIG. 1 illustrates an example of a system configured to navigate visualdata associated with a point of interest.

FIG. 2 illustrates an example of the data store of FIG. 1 in moredetail.

FIG. 3 illustrates an example of the server of FIG. 1 in more detail.

FIG. 4 illustrates an example process by which visual data associatedwith a point of interest may be navigated.

FIGS. 5A-5B illustrate example visual data associated with a point ofinterest.

FIG. 6 illustrates an example imagery graph.

FIGS. 7A-7F illustrate example click targets within visual dataassociated with a point of interest.

FIG. 8 conceptually illustrates an example electronic system with whichsome implementations of the subject technology are implemented.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology may bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, it will be apparent that the subjecttechnology is not limited to the specific details set forth herein andmay be practiced without these specific details. In some instances,structures and components are shown in block diagram form in order toavoid obscuring the concepts of the subject technology.

The subject technology is related to navigating visual data associatedwith a point of interest. In one example implementation, a serverprovides a first image of the point of interest for display at a clientcomputing device. The first image corresponds to an image node within animagery graph. The first image includes one or more points of interest(e.g., the Capitol Building or the Washington Monument in Washington,D.C.). The first image may be a member of a first imagery set (e.g.,street view images). The server receives an indication of a selection ofa point of interest within the first image. For example, the end-user ofthe client computing device may click on one or more pixels on a pointof interest (e.g., on a wall of the Capitol Building). Responsive to theindication of the selection of the point of interest, the serverselects, from image nodes associated with the point of interest withinthe imagery graph, a second image node. The second image node maycorrespond to an image of the point of interest, for example, an imagetaken from a viewpoint corresponding to the location on the first imagewhere the end-user selected the point of interest (e.g., the wall of theCapitol Building). The server provides the image corresponding to thesecond image node for display on the client computing device. The imagecorresponding to the second image node may be a member of a secondimagery set (e.g., user images from photograph sharing services)different from the first imagery set.

In some aspects, the imagery graph includes different types of images(e.g., street view panoramas, aerial images, satellite images, userimages from photograph sharing services, or images from videos) frommultiple imagery sets (e.g., imagery sets having sources including, forexample, public images within social networking services, images withinmapping services, public images within photograph sharing services,images from web-based encyclopedias, etc.). Each image within theimagery graph is associated with an image node. The image nodes areconnected in the imagery graph based on geographic location data of theassociated the image, as well as pose information (e.g., position,orientation, or field of view) associated with the image. However, poseinformation may not be available for some of the images in the imagerygraph.

Advantageously, in some implementations of the subject technology, auser is able to more easily navigate and interact with visual data of apoint of interest from multiple imagery sets (e.g., street viewpanoramas, aerial images, satellite images, user images from photographsharing services, or images from videos). The user navigation is basedon the three-dimensional characteristics of the point of interest suchthat the user is able to familiarize him/herself with thethree-dimensional characteristics of the point of interest, and how thethree-dimensional characteristics relate to images within the imagerysets.

FIG. 1 illustrates an example of a system 100 configured to navigatevisual data associated with a point of interest. As shown, the system100 includes a data store 110, a server 120, and a client computingdevice 130. The data store 110, server 120, and client computing device130 may be configured to communicate with one another via a network 140.The network 140 may include the Internet, an intranet, a local areanetwork, a wide area network, a wired network, a wireless network, or avirtual private network (VPN).

The data store 110 may store one or more imagery graphs that includeimages of one or more points of interest. The images may be frommultiple imagery sets (e.g., street view panoramas, aerial images,satellite images, user images from photograph sharing services, orimages from videos). The data store 110 may include a single machine,multiple machines, a single processor system, or a multi-processorsystem. One example of the data store 110 is described in more detail inconjunction with FIG. 2 below.

The server 120 may include a module to navigate visual data associatedwith a point of interest that is stored within the data store 110 orother sources. The server 120 may be implemented as a single machinewith a single processor, a multi-processor machine, or a server farmincluding multiple machines with multiple processors. One example of theserver 120 is described in more detail in conjunction with FIG. 3 below.

The client computing device 130 may be a laptop computer, a desktopcomputer, a mobile phone, a personal digital assistant (PDA), a tabletcomputer, a netbook, a television with one or more processors embeddedtherein or coupled thereto, a physical machine, or a virtual machine.The client computing device 130 may include one or more of a keyboard, amouse, a display, or a touch screen. The client computing device 130 mayalso include a web browser configured to display a webpage configured tonavigate visual data associated with a point of interest or include aspecial-purpose application (e.g., a mobile phone application or tabletcomputer application) for navigating visual data associated with a pointof interest. While only one client computing device 130 is illustratedin FIG. 1, the subject technology may be implemented in conjunction withmultiple client computing devices 130.

FIG. 2 illustrates an example of the data store 110 in more detail.

As shown, the data store 110 includes a processor 202, a networkinterface 204, and a memory 206. The processor 202 is configured toexecute computer instructions that are stored in a computer-readablemedium, for example, the memory 206. The processor 202 may be a centralprocessing unit (CPU). While only one processor 202 is illustrated, thedata store 110 may include multiple processors. The network interface204 is configured to allow the data store 110 to transmit and receivedata in a network, e.g., network 140 of FIG. 1. The network interface204 may include one or more network interface cards (NICs). The memory206 may store data or instructions. The memory 206 may include a cachememory unit, a storage memory unit, or any other memory unit. Asillustrated, the memory 206 includes a data structure representing animagery graph 208.

While the imagery graph 208 is illustrated as being represented in aunified imagery graph—a single data structure including multiplecomponents (e.g., image nodes 210)—the imagery graph 208 may,alternatively, be represented using multiple data structures, each ofwhich includes at least a portion of the data in the imagery graph 208.The imagery graph 208 includes image nodes 210 from multiple imagerysets 220. The imagery graph 208 also includes representation ofgeospatial relationships between the images corresponding to the imagenodes 210. The geospatial relationships are defined based on geographiclocation data of the associated the image, as well as pose information(e.g., position, orientation, or field of view) associated with theimage. However, pose information may not be available for some of theimages in the imagery graph 208. Each imagery set 220 may include one ormore image nodes 210. Example imagery sets 220 include street viewpanoramas, aerial images, satellite images, user images from photographsharing services, or images from videos. While imagery set 220.1 isillustrated as including one image node 210.1, and imagery set 220.2 isillustrated as including two image nodes 210.2-3, the subject technologymay be implemented with imagery sets that include multiple image nodes,for example, 50 image nodes, 500 image nodes, or 5000 image nodes.Furthermore, while only two imagery sets 220.1-2 are illustrated, thesubject technology may be implemented with any number of imagery sets.The imagery sets 220 may include different types of images. For example,one imagery set 220 may include aerial images and satellite images,while another imagery set may include street view panoramas, and yetanother imagery set may include user images from a photograph or videosharing service.

As set forth above, a unified imagery graph may include, among otherthings, a single data structure that includes multiple components (e.g.,image nodes 210) for representing images from multiple different imagesets (e.g., public images in a social networking service or anothersource, images in a mapping service, images in a web-based encyclopedia,etc.). The unified imagery graph may include the images within the datastructure or the unified imagery graph may include links (e.g., uniformresource indicators (URIs) or references to locations in memory) to theimages in the unified imagery graph. In some aspects, a unified imagerygraph is associated with a specific geographic location (e.g., the WhiteHouse in Washington, D.C.) and includes multiple images of thegeographic location from multiple different image sets. The unifiedimagery graph includes nodes representing the images and edgesrepresenting connections between the images (e.g., expanding ornarrowing a viewing angle, moving a camera position from one point toanother, etc.).

As set forth above, the imagery graph 208 (e.g., the unified imagerygraph) can include public images from multiple sources, for example,social networking service(s) or photograph sharing service(s). Imagesare not included in the imagery graph 208 without receiving appropriatepermissions from a user who uploaded the image to the network, server,or data repository. Visibility of an image may be controlled by the userwho uploaded the image (e.g., the user may restrict visibility tohis/her social contacts in the social networking service). Furthermore,unless appropriate permissions are received from a person depicted inthe image, the face of the person in the image is blurred to protecthis/her privacy.

As illustrated, each image node 210 includes an image 212. The image 212may be, for example, a photograph taken by a camera, an image derivedbased on a three-dimensional model, or a shot from a video. In someaspects, the image is associated with a geographic location 214, aviewpoint orientation 216, and a field of view 218. In some aspects, oneor more of the geographic location 214, the viewpoint orientation 216,or the field of view 218 for the image is unavailable. Images areorganized in the imagery graph 208 based on their geographiclocation(s), viewpoint orientation(s), or field(s) of view. Thegeographic location 214 may be based on a geographic tag within theimage. The geographic location 214 may be represented as a latitude andlongitude or as a street address. The geographic location 214 may becoupled with an accuracy radius (e.g., 100 meters). The geographiclocation may correspond to the precise two-dimensional orthree-dimensional position of the optical center of the image. Theviewpoint orientation 216 may include a direction in which a viewpointis directed. For example, if a photograph is taken while the lens of acamera is turned north, the viewpoint orientation 216 may be north. Theviewpoint orientation 216 may be expressed as a direction (e.g.,southeast) or as an angle (e.g., 135 degrees east of north/clockwisefrom north). The viewpoint orientation may have three angles (pan, tilt,and yaw) or one or more of these three angles. The field of view 218 maybe expressed as an angle covered by a camera, e.g., an angle between aleft or top edge of a photograph, a camera lens, and a right or bottomedge of a photograph. The field of view may be expressed as an angle(e.g., 45 degrees) or as multiple angles (e.g., 30 degrees left to rightand 105 degrees top to bottom). In some aspects, the geographiclocation, viewpoint orientation, or field of view are obtained viacomputer vision techniques.

FIG. 3 illustrates an example of the server 120 in more detail.

As shown, the server 120 includes a processor 302, a network interface304, and a memory 306. The processor 302 is configured to executecomputer instructions that are stored in a computer-readable medium, forexample, the memory 306. The processor 302 may be a central processingunit (CPU). While only one processor 302 is illustrated, the server 120may include multiple processors. Furthermore, while the server 120 isillustrated as a single machine, the server 120 may include multiplemachines, e.g., within a server farm. The network interface 304 isconfigured to allow the server 120 to transmit and receive data in anetwork, e.g., network 140 of FIG. 1. The network interface 304 mayinclude one or more network interface cards (NICs). The memory 306 maystore data or instructions. The memory 306 may include a cache memoryunit, a storage memory unit, or any other memory unit. As illustrated,the memory 306 includes a server-side imagery provider module 308.

The server-side imagery provider module 308 is configured to navigatevisual data associated with a point of interest. The server-side imageryprovider module 308 includes instructions to provide a first image(e.g., image 212.1) for display. The first image corresponds to an imagenode (e.g., image node 210.1) within an imagery graph (e.g., imagerygraph 208). The first image may include one or more points of interest(e.g., the Capitol Building and the Washington Monument). Theserver-side imagery provider module 308 also includes instructions toreceive an indication of a selection of a predetermined region withinthe first image (e.g., a mouse click on the predetermined region). Thepredetermined region is associated with a position in the first imagethat corresponds to geospatial data (e.g., a geographic location, aviewpoint position, or a viewpoint orientation) associated with a secondimage node. The server-side imagery provider module 308 also includesinstructions to provide a second image (e.g., image 212.2) correspondingto the second image node for display in response to the indicatedselection of the predetermined region. While the server-side imageryprovider module 308 is illustrated in FIG. 3 as a single module, theserver-side imagery provider module 308 may also be implemented asmultiple modules with each of the multiple modules being configured tocarry out or provide for carrying out of all or a portion of theinstructions described above.

In some aspects, the server-side imagery provider module 308 isconfigured to interact with a client-side imagery viewer moduleexecuting on the client computing device 130. For example, theindication of the selection of the predetermined region may come fromthe client-side imagery viewer module executing on the client computingdevice 130. The first image or the second image may be displayed via theclient-side imagery viewer module executing on the client computingdevice 130. The server-side imagery provider module 308 provides, to theclient-side imagery viewer module, image(s) along with navigationcontrols, which allow a user of the client computing device 130 toselect and receive additional imagery from the data store 110, via theserver 120, using the navigation controls. Example navigation controlsare discussed below in conjunction with FIGS. 5A-5B and FIGS. 7A-7F. Theclient-side imagery viewer module may be either a standalone application(e.g., a mobile phone application, a tablet computer application, or anapplication executing on a laptop or desktop computer) or a component ofan application (e.g., a component of a browser, a browser application,or a browser plug-in).

FIG. 4 illustrates an example process 400 by which visual dataassociated with a point of interest may be navigated.

The process 400 begins at step 410, where the server (e.g., server 120,by operation of the server-side imagery provider module 308) provides afirst image for display on a client computing device (e.g., clientcomputing device 130). The first image (e.g., image 212.1) correspondsto an image node (e.g., image node 210.1) within an imagery graph (e.g.,imagery graph 208). The imagery graph includes images from multipledifferent image types (e.g., photographs, depth-enhanced images,panoramic images, aerial images, satellite images, or video images) andimages from multiple different image sets (e.g., public images in asocial networking service, images in a mapping service, images in aweb-based encyclopedia, etc.). Each The first image may include one ormore points of interest. The imagery graph may include image nodes. Eachimage node in the imagery graph is associated with an image andgeospatial data (e.g., geographic location, viewpoint orientation, orfield of view) for the image node.

The imagery graph includes multiple image nodes. In some aspects, eachimage node in the imagery graph includes a corresponding image andcorresponding geographic location. The corresponding image is associatedwith a viewpoint position relative to a depicted point of interest, aviewpoint orientation, and a field of view. If an image includes a pointof interest having a known geographic location (e.g., the White Househas a known geographic location, which may be represented as a streetaddress or a latitude and longitude) and the image is associated with ageographic location (e.g., a geographic location tag), the viewpointorientation may be determined by comparing the geographic locationassociated with the image and the known geographic location of the pointof interest. For example, if a photograph of the White House is takenfrom a geographic location that is known to be west of the White House,it may be determined that the viewpoint orientation is facing east,toward the White House. Alternatively, the viewpoint orientation may bestored in association with the image (e.g., in a tag coupled of theimage). In some aspects, the geographic location, viewpoint orientation,or field of view of image(s) in the imagery graph are obtained viacomputer vision techniques.

In step 420, the server receives an indication of a selection of aregion (e.g., one or more pixels) within the first image. For example,the end-user of the client computing device may click the mouse onpixels corresponding to the region or touch pixels corresponding to theregion on a touch screen. The region could correspond to all or aportion of a point of interest (e.g., a depiction of the Statue ofLiberty in a photograph). The region could correspond to geospatial datain the image (e.g., a point on the north side of the Statue of Libertycould correspond to viewing the Statue of Liberty from the north). Theselected region could correspond to a click target. For example, a firstimage corresponding to a front view of the White House may include aclick target (i.e., a region indicating that it can be selected) at theright corresponding to moving to a right-side view of the White House,and a click target at the door corresponding to moving to an interiorview of the White House. The click target could be represented by athree-dimensional arrow or by dotted lines indicating that the regionpointed to by the three-dimensional arrow or within the dotted lines canbe selected (e.g., clicked with a mouse or touched on a touch screen) bythe user.

In some examples, a click target may not initially be indicated at all.When a user hovers a cursor over the click target, an indication of theclick target (e.g., the three-dimensional arrow or the dotted lines) mayappear or the cursor may change shape (e.g., becoming larger or becomingthe three-dimensional arrow) to provide a notification to the user thata click target is available.

In step 430, the server determines whether the selected region isassociated with a position in the first image that corresponds togeospatial data associated with a second image node in the imagerygraph. Such a position in the first image that corresponds to geospatialdata associated with a second image may correspond to a click target. Ifso, the process 400 continues to step 440. If not, the process 400returns to step 410.

In step 440, the server provides a second image corresponding to thesecond image node for display in response to the selection of theregion. The second image node is associated with the position of theselected region on the first image. For example, if the selected regionincludes a portion of a point of interest, the second image could be animage of the point of interest. If the selected region is on the eastside of the point of interest, the second image could be a view from theeast of the point of interest. The second image may be of a differentimagery type than the first image. For example, the first image may bean aerial image, and the second image may be a street view panorama.Alternatively, the first image and the second image may be of the sameimagery type.

In some aspects, the click target is a user-interface element forselecting a predetermined position on an image. For example, a clicktarget may represent an image node in the imagery graph that isselectable from an image that is currently displayed. A cursor maytransform into a shape different form its usual shape (e.g., into athree dimensional arrow or an arrow larger than a typical cursor arrow)when placed over one of the click targets to communication informationto the user that the cursor is hovering over a click target, and thatadditional image(s) associated with the click target are available.Alternatively, the cursor is not transformed when hovering over theclick target, and click target(s) are represented via fixedthree-dimensional arrow(s) or dotted lines, for example, as shown inFIGS. 7A-7F.

In one example, the first image corresponding to the first image node(e.g., image node 210.1) is from a first imagery set (e.g., 220.1)within the multiple imagery sets and the second image corresponding tothe second image node (e.g., image node 210.2) is from a second imageryset (e.g., imagery set 220.2) within the multiple imagery sets. Forexample, the first image may correspond to satellite image from asatellite imaging service, and the second image may correspond to ascreen shot from a video in a video sharing service. After step 440, theprocess 400 ends.

In one example aspect, instructions for implementing the steps 410-440in the process 400 may be stored in a memory of a computer and may beimplemented by one or more computers. The steps 410-440 in the process400 may be carried out in series. Alternatively, two or more of thesteps 410-440 in the process 400 may be carried out in parallel.

FIGS. 5A-5B illustrate example visual data 500A and 500B associated witha point of interest.

In one example aspect, image 500A of FIG. 5A may be presented to a userof a client computing device, for example, in response to a search forimages of the Capitol Building, images of the Washington Monument, orimages of landmarks in Washington, D.C. Image 500A corresponds to animage node in an imagery graph (e.g., imagery graph 208). Image 500A maybe an aerial photograph. Image 500A may include two points of interest:the Capitol Building 504A and the Washington Monument 506A. A user mayselect one of the points of interest 504A or 506A, for example, by usingthe cursor 502A. As shown in FIG. 5A, the user is in the process ofselecting the Capitol Building 504A on one side (e.g., the front side)of the Capitol Building 504A using the cursor 502A, which may beassociated with a mouse. In an alternative implementation, the user mayselect a point of interest 504A or 506A by touching the point ofinterest 504A or 506A with his/her finger on a touch screen.

In response to the user's selection of the Capitol Building 504A on theone side of the Capitol Building 504A, image 500B of FIG. 5B may bepresented to the user of the client computing device. The server mayassociate the user's selection of the Capitol Building 504A with animage node in the imagery graph corresponding to image 500B, as image500B is an image of the Capitol Building from a position correspondingto the position of the cursor 502A in the image 500A of FIG. 5A. Image500B may be a user photograph or a street view panorama. As shown, image500B includes a view of the Capitol Building 504B (which corresponds tothe Capitol Building 504A) from the one side of the Capitol Building,which was selected by the cursor 502A in image 500A of FIG. 5A.

FIG. 6 illustrates an example imagery graph 600.

As shown, the imagery graph 600 indicates a position of a point ofinterest 602 (indicated with a star) and a position of a street 604. Theimagery graph 600 also includes street view panoramas 606, 608, 610,612, 614 taken on the street 604 and user photographs 616, 618, 620, and622 of the point of interest 602. The street 604 runs from east to westand the point of interest 602 is positioned north of the street 604.

In one example aspect, a user of a client computing device may beviewing the point of interest 602 from a north-facing street viewpanorama 610. While looking at the north-facing street view panorama610, the user may become interested in what the point of interest 602looks like from the right (east) side. Accordingly, the user may selecta point on the right (east) side of the point of interest 602 in thestreet view panorama 610. In response, the server may provide the userphotograph 618 of the point of interest 602 taken from the east side ofthe point of interest 602 for display.

In another example aspect, while viewing the north-facing street viewpanorama 610 of the point of interest 602, the user may want to see amore detailed or zoomed-in view of the point of interest 602.Accordingly, the user may select a point on the point of interest 602about which he/she wants more detail (e.g., a door or window of abuilding, a head of a statue, etc.). In response, the server may providethe user photograph 620 of the point of interest 602, which includes amore detailed or zoomed-in view of the point of interest 602, fordisplay.

FIGS. 7A-7F illustrate example click targets within visual data 700A,700B, 700C, 700D, 700E, and 700F associated with a point of interest.

As shown, image 700A includes click targets 702A and 706A. The clicktargets 702A and 706A are indicated by three-dimensional arrows 704A and708A, respectively. The three-dimensional arrows 704A or 708A mayindicate a viewpoint orientation for an image to be displayed inresponse to the user selecting the click targets 702A or 706A,respectively. For example, if click target 702A is selected, a bottomleft wall of the building in image 700A may be displayed in greaterdetail. If click target 706A is selected, a bottom right wall of thebuilding in image 700A may be displayed in greater detail. The imagespresented responsive to selections the click targets 702A and 706A maybe from different imagery types from one another and/or from a differentimagery type than image 700A. The imagery types may include photographs,depth-enhanced images, panoramic images, aerial images, satelliteimages, or video images. For example, image 700A may correspond to anaerial photograph, while images corresponding to the click targets 702Aor 706A may be panoramic images or photographs. Furthermore, the image700A may be from a different imagery set than the images associated withthe click targets 702A or 706A. The images associated with the clicktargets 702A and 706A may be from different imagery sets. Imagery setsmay include public images from a social networking service, images froma mapping service, web-based encyclopedia images, etc. As shown in FIGS.7A-7F, click targets are indicated by three-dimensional arrows. However,other graphics can be used, in place of or in addition to thethree-dimensional arrows, to indicate click targets.

Image 700B includes click targets 702B, 706B, and 710B indicated bythree-dimensional arrows 704B, 708B, and 712B, respectively. Thethree-dimensional arrows 704B, 708B, and 712B may indicate directions ofmotion from the image 700B to the images resulting from selecting theclick targets 702B, 706, or 710B, respectively. For example, clicktarget 706B, indicated by forward-pointing arrow 708B, may correspond tomoving forward. Click target 702B, indicated by arrow 704B, maycorrespond to moving forward and slightly to the left, in the directionindicated by arrow 704B. Click target 710B, indicated by right-pointingarrow 712B, may correspond to moving to the right. Image 700B may be astreet view panorama, images corresponding to the click targets 702B and710B may be photographs, and an image corresponding to click target 706Bmay be an aerial image.

Image 700C includes click target 702C and three-dimensional arrow 704C.Click target 702C may correspond to a zoomed-in view of the arc point ofinterest, as indicated by arrow 704 and the region 702C. Image 700C maybe a street view panorama, while an image corresponding to click target702C may be a depth-enhanced image.

Image 700D includes click target 702D and three-dimensional arrow 704D.The three-dimensional arrow 704D points into the building point ofinterest, indicating that click target 702D corresponds to viewing theinterior of the building point of interest. In order for the server toprovide images of the interior of the building point of interest, theowner and the occupant of the building have provided proper permissionsfor the server and data store to store such images and to provide suchimages to users of client computing devices. The image 700D may be adepth-enhanced image, while an image corresponding to click target 702Dmay be a photograph.

Image 700E includes a click target 702E for viewing a scene to the rightof the scene in image 700E, as indicated by the three-dimensional arrow704E and a click target 706E for viewing an avenue scene behind thescene in image 700E, as indicated by the backward-pointingthree-dimensional arrow 708E. Image 700E may be from a set ofdepth-enhanced images, while the images associated with the clicktargets 702E and 706E may be public photographs stored within a socialnetworking service or street view panoramas stored within a mappingservice.

Image 700F includes a scene within a room. Click target 702F is forviewing a scene in an adjacent room to the left, as indicated bythree-dimensional arrow 704F. Click target 712F is for viewing a scenein an adjacent room to the right, as indicated by three-dimensionalarrow 710F. Click target 706F is for viewing the ceiling of the room ingreater detail, as indicated by three-dimensional arrow 708F. The image700F may be from a different image set or a different image type fromthe images associated with the click targets 702F, 706F, and 712F.

FIG. 8 conceptually illustrates an electronic system 800 with which someimplementations of the subject technology are implemented. For example,one or more of the data store 110, the server 120, or the clientcomputing device 130 may be implemented using the arrangement of theelectronic system 800. The electronic system 800 can be a computer(e.g., a mobile phone, PDA), or any other sort of electronic device.Such an electronic system includes various types of computer readablemedia and interfaces for various other types of computer readable media.Electronic system 800 includes a bus 805, processing unit(s) 810, asystem memory 815, a read-only memory 820, a permanent storage device825, an input device interface 830, an output device interface 835, anda network interface 840.

The bus 805 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 800. For instance, the bus 805 communicativelyconnects the processing unit(s) 810 with the read-only memory 820, thesystem memory 815, and the permanent storage device 825.

From these various memory units, the processing unit(s) 810 retrievesinstructions to execute and data to process in order to execute theprocesses of the subject technology. The processing unit(s) can be asingle processor or a multi-core processor in different implementations.

The read-only-memory (ROM) 820 stores static data and instructions thatare needed by the processing unit(s) 810 and other modules of theelectronic system. The permanent storage device 825, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system800 is off. Some implementations of the subject technology use amass-storage device (for example a magnetic or optical disk and itscorresponding disk drive) as the permanent storage device 825.

Other implementations use a removable storage device (for example afloppy disk, flash drive, and its corresponding disk drive) as thepermanent storage device 825. Like the permanent storage device 825, thesystem memory 815 is a read-and-write memory device. However, unlikestorage device 825, the system memory 815 is a volatile read-and-writememory, such a random access memory. The system memory 815 stores someof the instructions and data that the processor needs at runtime. Insome implementations, the processes of the subject technology are storedin the system memory 815, the permanent storage device 825, or theread-only memory 820. For example, the various memory units includeinstructions for navigating visual data associated with a point ofinterest in accordance with some implementations. From these variousmemory units, the processing unit(s) 810 retrieves instructions toexecute and data to process in order to execute the processes of someimplementations.

The bus 805 also connects to the input and output device interfaces 830and 835. The input device interface 830 enables the user to communicateinformation and select commands to the electronic system. Input devicesused with input device interface 830 include, for example, alphanumerickeyboards and pointing devices (also called “cursor control devices”).Output device interfaces 835 enables, for example, the display of imagesgenerated by the electronic system 800. Output devices used with outputdevice interface 835 include, for example, printers and display devices,for example cathode ray tubes (CRT) or liquid crystal displays (LCD).Some implementations include devices for example a touchscreen thatfunctions as both input and output devices.

Finally, as shown in FIG. 8, bus 805 also couples electronic system 800to a network (not shown) through a network interface 840. In thismanner, the electronic system 800 can be a part of a network ofcomputers (for example a local area network (“LAN”), a wide area network(“WAN”), or an Intranet, or a network of networks, for example theInternet. Any or all components of electronic system 800 can be used inconjunction with the subject technology.

The above-described features and applications can be implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storageor flash storage, for example, a solid-state drive, which can be readinto memory for processing by a processor. Also, in someimplementations, multiple software technologies can be implemented assub-parts of a larger program while remaining distinct softwaretechnologies. In some implementations, multiple software technologiescan also be implemented as separate programs. Finally, any combinationof separate programs that together implement a software technologydescribed here is within the scope of the subject technology. In someimplementations, the software programs, when installed to operate on oneor more electronic systems, define one or more specific machineimplementations that execute and perform the operations of the softwareprograms.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

These functions described above can be implemented in digital electroniccircuitry, in computer software, firmware or hardware. The techniquescan be implemented using one or more computer program products.Programmable processors and computers can be included in or packaged asmobile devices. The processes and logic flows can be performed by one ormore programmable processors and by one or more programmable logiccircuitry. General and special purpose computing devices and storagedevices can be interconnected through communication networks.

Some implementations include electronic components, for examplemicroprocessors, storage and memory that store computer programinstructions in a machine-readable or computer-readable medium(alternatively referred to as computer-readable storage media,machine-readable media, or machine-readable storage media). Someexamples of such computer-readable media include RAM, ROM, read-onlycompact discs (CD-ROM), recordable compact discs (CD-R), rewritablecompact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM,dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g.,DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SDcards, micro-SD cards, etc.), magnetic or solid state hard drives,read-only and recordable Blu-Ray® discs, ultra density optical discs,any other optical or magnetic media, and floppy disks. Thecomputer-readable media can store a computer program that is executableby at least one processing unit and includes sets of instructions forperforming various operations. Examples of computer programs or computercode include machine code, for example is produced by a compiler, andfiles including higher-level code that are executed by a computer, anelectronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some implementations areperformed by one or more integrated circuits, for example applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some implementations, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium” and “computer readable media” are entirelyrestricted to tangible, physical objects that store information in aform that is readable by a computer. These terms exclude any wirelesssignals, wired download signals, and any other ephemeral signals.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

The subject matter described in this specification can be implemented ina computing system that includes a back end component, e.g., as a dataserver, or that includes a middleware component, e.g., an applicationserver, or that includes a front end component, e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the subject matter described inthis specification, or any combination of one or more such back end,middleware, or front end components. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”), aninter-network (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someaspects of the disclosed subject matter, a server transmits data (e.g.,an HTML page) to a client device (e.g., for purposes of displaying datato and receiving user input from a user interacting with the clientdevice). Data generated at the client device (e.g., a result of the userinteraction) can be received from the client device at the server.

It is understood that any specific order or hierarchy of steps in theprocesses disclosed is an illustration of example approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of steps in the processes may be rearranged, or that allillustrated steps be performed. Some of the steps may be performedsimultaneously. For example, in certain circumstances, multitasking andparallel processing may be advantageous. Moreover, the separation ofvarious system components illustrated above should not be understood asrequiring such separation, and it should be understood that thedescribed program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts.

Various modifications to these aspects will be readily apparent, and thegeneric principles defined herein may be applied to other aspects. Thus,the claims are not intended to be limited to the aspects shown herein,but is to be accorded the full scope consistent with the languageclaims, where reference to an element in the singular is not intended tomean “one and only one” unless specifically so stated, but rather “oneor more.” Unless specifically stated otherwise, the term “some” refersto one or more. Pronouns in the masculine (e.g., his) include thefeminine and neuter gender (e.g., her and its) and vice versa. Headingsand subheadings, if any, are used for convenience only and do not limitthe subject technology.

A phrase, for example, an “aspect” does not imply that the aspect isessential to the subject technology or that the aspect applies to allconfigurations of the subject technology. A disclosure relating to anaspect may apply to all configurations, or one or more configurations. Aphrase, for example, an aspect may refer to one or more aspects and viceversa. A phrase, for example, a “configuration” does not imply that suchconfiguration is essential to the subject technology or that suchconfiguration applies to all configurations of the subject technology. Adisclosure relating to a configuration may apply to all configurations,or one or more configurations. A phrase, for example, a configurationmay refer to one or more configurations and vice versa.

What is claimed is:
 1. A computer-implemented method for navigating animagery graph, the method comprising: providing a first image fordisplay, wherein the first image corresponds to a first image nodewithin an imagery graph, wherein the imagery graph comprises image nodescorresponding to images from a plurality of different imagery types, andwherein each image node in the imagery graph is associated withgeospatial data; receiving an indication of a selection of apredetermined region within the first image, wherein the predeterminedregion is associated with a position in the first image that correspondsto geospatial data associated with a second image node within theimagery graph; and providing a second image corresponding to the secondimage node for display in response to the indication of the selection ofthe predetermined region.
 2. The method of claim 1, wherein theplurality of different imagery types comprise two or more of:photographs, depth-enhanced images, panoramic images, aerial images,satellite images, or video images.
 3. The method of claim 2, wherein thefirst image is from a different imagery type than the second image. 4.The method of claim 1, wherein the predetermined region comprises one ormore pixels.
 5. The method of claim 1, wherein the predetermined regioncomprises a click target.
 6. The method of claim 1, wherein thepredetermined region comprises at least a portion of a point of interestdisplayed within the first image.
 7. The method of claim 6, wherein thesecond image comprises an image of the point of interest.
 8. The methodof claim 7, wherein a viewpoint orientation of the second imagecorresponds to viewing the point of interest from a positioncorresponding to the predetermined region on the first image.
 9. Themethod of claim 1, wherein the geospatial data comprises one or more ofa geographic location, a viewpoint orientation, or a field of view. 10.The method of claim 1, further comprising: receiving indicia of a cursorbeing hovered within the predetermined region; and providing anotification that the second image is available responsive to theindicia of the cursor being hovered within the predetermined region. 11.The method of claim 10, wherein the notification comprises atransformation of a shape of the cursor.
 12. A computer-readable mediumcomprising instructions which, when executed by one or more computers,cause the one or more computers to implement a method, the methodcomprising: providing a first image for display, wherein the first imagecorresponds to a first image node within an imagery graph, wherein theimagery graph comprises plural image nodes, and wherein each image nodein the imagery graph is associated with geospatial data; receiving anindication of a selection of one or more pixels within the first image,wherein the one or more pixels are associated with a position in thefirst image that corresponds to geospatial data associated with a secondimage node within the imagery graph; and providing a second imagecorresponding to the second image node for display in response to theindication of the selection of the one or more pixels.
 13. Thecomputer-readable medium of claim 12, wherein the plural image nodescorrespond to images from a plurality of different imagery types. 14.The computer-readable medium of claim 13, wherein the plurality ofdifferent imagery types comprise two or more of: photographs,depth-enhanced images, panoramic images, aerial images, satelliteimages, or video images.
 15. The computer-readable medium of claim 12,wherein the one or more pixels comprise a click target.
 16. Thecomputer-readable medium of claim 12, wherein the one or more pixelscomprise at least a portion of a point of interest displayed within thefirst image.
 17. The computer-readable medium of claim 16, wherein thesecond image comprises an image of the point of interest.
 18. Thecomputer-readable medium of claim 17, wherein a viewpoint orientation ofthe second image corresponds to viewing the point of interest from aposition corresponding to the one or more pixels on the first image. 19.The computer-readable medium of claim 12, wherein the geospatial datacomprises one or more of a geographic location, a viewpoint orientation,or a field of view.
 20. A system comprising: one or more processors; anda memory comprising: one or more data structures representing a unifiedimagery graph comprising image nodes corresponding to images from aplurality of different imagery types, wherein each image node in theunified imagery graph is associated with a geographic location, aviewpoint orientation, and a field of view; and instructions which, whenexecuted by the one or more processors, cause the one or more processorsto implement a method, the method comprising: providing a first imagefor display, wherein the first image corresponds to a first image nodewithin the unified imagery graph, and wherein the first image has afirst imagery type; receiving an indication of a selection of apredetermined region within the first image, wherein the predeterminedregion is associated with a position in the first image that correspondsto a geographic location, a viewpoint orientation, or a field of viewassociated a second image node within the unified imagery graph; andproviding a second image corresponding to the second image node fordisplay in response to the indication of the selection of thepredetermined region, wherein the second image has a second imagery typedifferent from the first imagery type.
 21. The system of claim 20,wherein the plurality of different imagery types comprise two or moreof: photographs, depth-enhanced images, panoramic images, aerial images,satellite images, or video images.
 22. The system of claim 20, whereinthe unified imagery graph comprises images from a plurality of differentimage sets.
 23. The system of claim 22, wherein the plurality ofdifferent image sets comprise two or more of: public images in a socialnetworking service, images in a mapping service, or images in aweb-based encyclopedia.